DOCUMENT RESUME 



ED 475 625 



PS 031 267 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



AVAILABLE FROM 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Mott, Michael Seth; Etsler, Cynthia; Drumgold, Deondra 
Applying an Analytic Writing Rubric to Children’s Hpermedia 
"Narratives” . 

2003-00-00 

18p.; In: Early Childhood Research & Practice: An Internet 
Journal on the Development, Care, and Education of Young 
Children, Spring 2003; see PS 031 261. Published bi-annually. 
For full text: http://ecrp.uiuc.edu/v5nl/mott.html. 

Reports - Evaluative (142) 

EDRS Price MFOl/PCOl Plus Postage. 

^Elementary School Students; Evaluation Methods; Grade 2; 
Grade 3; ^Hypermedia; ^Reliability ; ^Scoring Rubrics; 
^Validity 



ABSTRACT ' 

In an effort designed to guide and improve the assessment of 
a newly developed writing environment, the reliability and developmental and 
concurrent validity of a previously validated rubric developed for pen-and- 
paper-created narratives. Writing What You Read (WWYR) , was determined when 
applied to hypermedia-authored narratives of children in second and third 
grade. Children (n = 60) from four intact classrooms produced hypermedia 
narratives (text, audio, graphic, and video elements) over a 4-month period 
in a school-based computer laboratory. Raters (n = 5) with knowledge of the 
teaching of process writing and use of hypermedia software judged the 
hypermedia narrative productions. Raters judged all students’ (n = 60) 
hypermedia narrative productions individually without resolving differences 
through discussion. Two analyses were used to determine reliability: 
percentages of agreement and Pearson correlations. Percentages of agreement 
for the WWYR rubric averaged across 10 pairs of raters found high percentages 
of agreement among raters (.70 for ±0 and .99 for ±1) . Pearson correlations 
averaged across 10 pairs of raters found acceptable interrater reliability 
for four of the five subscales. For the five subscales (Theme, Character, 
Setting, Plot, and Communication), the r values were .59, .55, .49, .50, 

and .50, respectively. Developmental validity of the WWYR scores was examined 
with one-way BANOVA to evaluate the WWYR scores of children grouped as low, 
medium, or high ability based on their Iowa Test of Basic Skills (ITBS) 
National Percentile Rank for Literacy Skill. Evidence for the developmental 
validity of the WWYR scores was supported across the three ability groups, F 
(2, 36) = 2.59, p <.01. Concurrent validity was examined through 
correlational analysis between students’ mean WWYR score and ITBS score. 
Scores from the two measures were positively correlated,- r = .83, p < .01, 
providing evidence of the sensitivity of the WWYR assessment to measure the 
developmental literacy competency of the third-grade students. Results 
support teachers’ use of a validated rubric developed for pen-and-paper- 
created narratives applied to hypermedia narratives, despite additional 
visual -and audio narrative elements inherent to hypermedia. Implications for 
literacy teaching and learning with hypermedia address core questions about 
the similarities and differences between written textual expression and 
visual and verbally recorded expression. A revised and expanded WWYR rubric 
is proposed to begin to address these core questions generated by teachers’ 
use of the hypermedia writing environment. (Author) 
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Abstract 





In an effort designed to guide and improve the assessment of a newly developed writing environment, the reliability 
and developmental and concurrent validity of a previously validated rubric developed for pen-and-paper-created 
narratives, Writing What You Read (WWYR), was determined when applied to hypermedia-authored narratives of 
children in second and third grade. Children (n = 60) from four intact classrooms produced hypermedia narratives 
(text, audio, graphic, and video elements) over a 4-month period in a school-based computer laboratory. Raters (n = 5) 
with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative 
productions. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving 
differences through discussion. Two analyses were used to determine reliability: percentages of agreement and 
Pearson correlations. Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high 
percentages of agreement among raters (.70 for ±0 and .99 for ±1). Pearson correlations averaged across 10 pairs of 
raters found acceptable interrater reliability for four of the five subscales. For the five subscales (Theme, Character, 
Setting, Plot, and Communication), the r values were .59, .55, .49, .50, and .50, respectively. Developmental validity 
of the WWYR scores was examined with one-way MANOVA to evaluate the WWYR scores of children grouped as 
low, medium, or high ability based on their Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy 
Skill. Evidence for the developmental validity of the WWYR scores was supported across the three ability groups, F 
(2, 36) = 2.59, p <. 01. Concurrent validity was examined through correlational analysis between students' mean 
WWYR score and ITBS score. Scores from the two measures were positively correlated, r = .83, p < .01, providing 
evidence of the sensitivity of the WWYR assessment to measure the developmental literacy competency of the third- 
grade students. Results support teachers' use of a validated rubric developed for pen-and-paper-created narratives 
applied to hypermedia narratives, despite additional visual and audio narrative elements inherent to hypermedia. 
Implications for literacy teaching and learning with hypermedia address core questions about the similarities and 
differences between written textual expression and visual and verbally recorded expression. A revised and expanded 
WWYR rubric is proposed to begin to address these core questions generated by teachers' use of the hypermedia 
writing environment. 





Introduction 

The very notion of ’’writing" is increasingly being transformed by new digital computer technologies in 
society, homes, businesses, and schools. Today, one's ability to represent thought electronically is more 
important than ever. The extreme pace of change in society dictates that early childhood educators 
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consider how to facilitate children in learning to express themselves digitally in computer environments in 
developmentally appropriate ways. This article addresses the curriculum, instruction, and assessment of 
children’s written expression in hypermedia, a computer environment that supports text, audio, video, and 
graphics. The hypermedia curriculum and instruction are described, and a technical analysis of a proposed 
assessment is applied and discussed. 



Emerging Trends 

Prior research has addressed the reliability and validity of a narrative rubric useful for both teaching and 
learning (value) and large-scale (utility) literacy assessment. As literacy curriculum and instruction 
continue to transform to include new computer-based learning environments, researchers must 
correspondingly develop and technically evaluate assessments for the new environments. Hypermedia 
represents a powerful electronic environment through which literacy expression can be facilitated by the 
manipulation of text, graphics, audio, and video elements, and this technology and integration are 
reflected in the new literacy K-12 standards (IRA and NCTE, 2001). The qualities of hypermedia that 
support higher-level cognitive processes such as synthesis, organization, evaluation, and reflexivity have 
been well documented (Yang, 1996; Mott, Sumrall, & Hodges, 1997). However, there is a glaring absence 
of ways to reliably and validly assess students’ hypermedia products. To address this absence, a narrative , 
process writing curriculum and instruction environment was merged with hypermedia. These terms are 
defined as follows: 

• Process writing curriculum and instruction: writing using discrete stages (brainstorming, drafting, 
conferencing, revising, editing, and publishing) administered via ”mini-lessons” applied to address 
the needs of the individual writers in whole-group instruction. See Graves (1983) for a detailed 
description. 

• Hypermedia: Hypermedia comprises two main components: (1) hyper: the ability to program 
electronic links, or hyperconnections, to connect information to any other Internet- based source or 
simply to link locally to a hard drive or diskette, and (2) media: the ability to manipulate multiple 
meaning-based symbol systems representing a variety of sources — text, graphics, audio, and video 
clips. 

This article addresses the reliability and developmental and concurrent validity of a previously validated 
narrative writing rubric. Writing What You Read (WWYR) (Wolf & Gearhart, 1994; Novak, Herman, & 
Gearhart, 1996), applied to hypermedia narratives created by students in grades 2 and 3 (see Table 1). The 
technical qualities of the process -oriented classroom rubric, valuable for teaching and learning on a day- 
to-day basis, are linked to the utility of the narrative rubric for measuring elementary students’ literacy 
competencies as identified via a validated large-scale instrument, the Iowa Test of Basic Skills — Literacy 
Competency. Thus, five main issues are addressed relating to rater judgments of elementary students’ 
hypermedia narratives (stories with text, graphics, audio, and video elements authored using HyperStudio 
hypermedia software): 

• Reliability of interrater judgments is examined using the WWYR of hypermedia narrative quality. 

• Developmental validity of the WWYR scores is examined with one-way MANOVA used to 
evaluate the WWYR scores of students grouped as low, medium, or high ability based on their 
Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy Skill. 

• Concurrent validity is examined through correlational analysis of students’ mean WWYR scores 
and ITBS scores. 

• Value of the WWYR for use as a teaching tool is summarized through reviewing the genesis of the 
rubric as demonstrated in its path from creation for pen-and-paper narratives to hypermedia 
narratives as evaluated in multiple studies. 

• Utility of the WWYR applied to students’ hypermedia narratives is addressed to reveal possible 
alignment between the innovative curriculum and instruction addressed in the current study with 
ITBS- and WWYR- identified literacy levels. 

Table 1 

Writing What You Read Narrative Rubric (Wolf & Gearhart, 1994) 
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Theme 


Character 


Setting 


Plot 


Communication 


Explicit-Implicit 


Flat-Round 


Backdrop-Essential 


Simple-Complex 


Context-bound 


Didactic- 

Revealing 


Static-Dynamic 


Simple- 

Multifunctional 


Static-Conflict 


Literal-Symbolic 


1: Not present or 
not developed 
through other 
narrative 
elements 


1: One or two flat, 
static characters, 
with little 
relationship 
between characters 


1: Backdrop setting 
with little or no 
indication of time or 
place ("There was a 
little girl. She liked 
candy.") 


1: One or two 
events with little 
or no conflict 
("Once there was 
a cat. The cat 
liked milk.") 


1: Writing bound to 
context (You have to 
be there) and often 
dependent on drawing 
and talk to clarify the 
meaning 


2: Meaning 
centered in a 
series of list-like 
statements ("I 
like my mom. 
And I like my 
dad. And I like 
my...") 


2: Some rounding, 
usually in physical 
description; 
relationship 
between characters 
is action driven 


2: Skeletal indication 
of time and place 
often held in past 
time ("Once there 
was..."); little 
relationship to other 
narrative elements 


2: Beginning 
sequence of 
events, but out- 
of-sync 
occurrences; 
events without 
problem; problem 
without 
resolution 


2: Beginning 
awareness of reader 
considerations; 
straightforward style 
and tone focused on 
getting the 
information out 


3: Beginning 
statement of 
theme, often 
explicit and 
didactic ("The 
mean witch 
chased the 
children and she 
shouldn't have 
done that.") 


3: Continued 
rounding in 
physical 
description, 
particularly 
stereotypical 
features ("wart on 
the end of her 
nose") 


3: Beginning 
relationship between 
setting and other 
narrative elements 
(futuristic setting to 
accommodate aliens 
and spaceships) 


3: Single, linear 
episode with 
clear beginning, 
middle, and end; 
the episode 
contains a 
problem, 
emotional 
response, action, 
and outcome 


3: Writer begins to 
make sense of 
explanations and 
transitions ("because" 
and "so"); literal style 
centers on description 


4: Beginning 
revelation of 
theme on both 
explicit and 
implicit levels 
through the more 
subtle things 
characters say 
and do 


4: Beginning 
insights into 
motivation and 
intention that drive 
the feeling and 
action of main 
characters often 
through limited 
omniscient point of 
view 


4: Setting becomes 
more essential to the 
development of the 
story in explicit ways: 
characters may 
remark on the setting, 
or the time and place 
may be integral to the 
plot 


4: Plot increases 
in complexity 
with more than 
one episode; each 
episode contains 
problem, 
emotional 
response, action, 
and outcome 


4: Increased 
information and 
explanation for the 
reader (linking ideas 
as well as episodes); 
words more carefully 
selected to suit the 
narrative's purpose 


5: Beginning use 
of secondary 
themes, often tied 
to overarching 
theme, but 
sometimes 
tangential 


5: Further rounding 
(in feeling and 
motivation); 
dynamic features 
appear in central 
characters and 
between characters 


5: Setting may serve 
more than one 
function, and the 
relationship between 
functions is more 
implicit and symbolic 


5: Stronger 
relationships 
between episodes 
(with resolution 
in one leading to 
a problem in the 
next) 


5: Some 

experimentation with 
symbolism 

(particularly figurative 
language), which 
shows reader 
considerations 


6: Overarching 
theme 

multilayered and 
complex; 
secondary themes 
integrally related 
to the primary 
themes 


6: Round, dynamic 
major characters 
through rich 
description of 
affect, intention, 
and motivation 


6: Setting fully 
integrated with the 
characters, action, 
and theme 


6: Overarching 
problem and 
resolution 
supported by 
multiple episodes 


6: Careful crafting of 
choices of story 
structure as well as 
vocabulary 
demonstrate 
considerate 
orchestration of all 
resources 
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Methods 

Children (n = 60) from four intact classrooms (two second-grade and two third-grade classrooms) 
produced hypermedia narratives over a 4-month period in a school-based computer laboratory equipped 
with 10 Windows-based microcomputers. Rater/(Teachers) (n = 5) with knowledge of the teaching of 
process writing and use of hypermedia software judged the hypermedia narrative productions. An 
interactive hypermedia software tutorial program was developed and used to train the teachers (n = 4) in 
the implementation of process writing techniques in conjunction with the use of hypermedia features as 
part of their elementary curriculum. Raters participated in a 3-hour training and rating session in a 
university computer laboratory equipped with five Power Macintosh microcomputers. Raters judged all 
students’ (n = 60) hypermedia narrative productions individually without resolving differences through 
discussion. 



Materials 

The WWYR rubric shown in Table 1 contained five evaluative scales designed to assess students' 
developing competencies in narrative writing: Theme, Character, Setting, Plot, and Communication. The 
vertical analytical evaluative scales (1-6 for each competency) were designed to enable teachers to make 
instructional decisions on specific narrative components needing reinforcement and were not intended as a 
method for assigning a numerical value to a narrative. Teachers merely had to shade a box in the rubric to 
denote where a child's narrative was along each competency. The ITBS (Linn & Wilson, 1990) Form J 
was used as a basic battery for grades K-9 and included language skills directly related to writing: word 
analysis, vocabulary, spelling, and reading comprehension. Reliability coefficients for Form J ranged 
from .70-.90 for the language skills components. Additionally, the ITBS met high standards of overall 
technical quality and has been a widely accepted standardized measure of cognitive skill. 



Hypermedia Narratives 

HyperStudio Presentation Software (Wagner, 1997-2001) was used to support the children in expressing 
themselves with text, audio, video, and graphic elements in their narrative productions (see Figures 1-4). 
Children composed on paper and computer with teachers structuring the process with Writing Workshop 
(Graves, 1983), a method for organizing writing into discrete and recursive stages. These stages are (1) 
brainstorming, (2) revising, (3) drafting, (4) peer conferencing, (5) editing, and (6) publishing (see Mott & 
Klomes, 2001, for a detailed description of a program similar to the program addressed in the current 
study). 
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BySallyM. 



[. Page' 1 



Figure I. HyperStudio narrative "page" with hypermedia elements: Grade 2. 

Elements shown are {}) graphics text, (2) paint bucket tool (blue), (3) line tool: freehand, (4) hyperconnection or link 

to "Page 1 " via button. 




Figure 2. Page with the following features: textboxAvord processed text with scrolling and a graphic (cat) inserted 

into Shape. 
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The next day M rs. N 
told me that I 
could bring my cat 
to scho ol if my 
mother came too. 



I wassooooo 



Figure 3. HyperStudio page. Note control of font, color, background color, and font size. 
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I decided that we 
would go to the 
playground near 
the climbing bars. 

Grey will be 

comfortable out 

1.1 



Back to Front 



j 












Figure 4. This page includes a scanned photograph. 
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HyperStudio was selected for this project for a number of reasons: (1) it supports high-end features such 
as video and animation; (2) it is relatively easy to use, even by young children (grades 1-3); (3) it contains 
a flexible interface; and (4) it is the most widely used multimedia/hypermedia platform in elementary 
education with over 100,000 users (Wagner, 1997-2001). HyperStudio contains several programming 
features that support children's hypermedia programming (see Figures 5-10). Programming instruction 
(and writing workshop) occurred through mini-lessons over a 4-month period. Figures 5 through 10 
contain examples from a mini-lesson. 




Figure 5 . This page contains the programming line tool 
The line tool was introduced to children during their first programming mini-lessons. 
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Figure 6. This page contains the hyperconnection programming tool. 
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The "hyper” programming tool, button-creation, enabled children to connect cards (narrative pages) to 
other cards. Button-creation supported linear links, (card 1 to 2 to 3), suitable for early childhood. 




Figure 7. This page contains the graphics tool. 

Children inserted graphics and scanned images from art representing many media: watercolor, acrylic 
paint, crayons, and colored pencils. 




Figure 8. This page contains the textbox tool (word processing). 

Writing was accomplished via the textbox tool that functions as a word processor with editing tools, 
cutting, pasting, and other word processor capabilities. 
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Figure 9. This page contains the digital audio deck. 



Children inserted audio clips, programmed via button-creation, to enrich their text. Audio clips were 
recorded by the author or downloaded from HyperStudio for special effects such as an alarm clock sound. 




Figure 10. This page contains video control. 

Video clips were inserted, programmed via button-creation, to supplement the narrative. One example was 
a video clip of Mars used in a science fiction narrative by a third-grade student. 
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Percentages of Agreement 

Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages 
of agreement among raters (.70 for ±0 and .99 for ±1) (see Table 2). The ±0 and ±1 percentages of 
agreement across 10 pairs of raters were higher than the ±0 and ±1 agreement levels found in both the 
Gearhart, Herman, Novack, and Wolf (1995) and Novak, Herman, and Gearhart (1996) WWYR reliability 
studies (compared in Table 3). The high percentages of agreement found in this study may be attributed to 
the raters' use of only the first three WWYR rubric evaluative subscale levels. The WWYR rubric 
contains six subscale levels that are developmentally sequenced according to the varied writing 
competencies of students in grades K-6. Because students in this study were in second and third grade, 
raters typically applied only levels 1, 2, and 3 out of five total levels. This narrow range of values 
independently applied by raters limited the number of choices. Hence, high percentages of agreement 
between raters would be expected based on the limited number of scale levels used. 

Table 2 

Percentages of Agreement for all Five Subscales of the WWYR Rubric 



WWYR Subscale 


±0 


±1 


n 


Theme 


.70 


.96 


60 


Character 


.78 


.99 


60 


Plot 


.73 


.99 


60 


Setting 


.67 


.99 


60 


Communication 


.68 


.99 


60 



Table 3 

Percentages of Agreement for the WWYR Rubric Averaged across All Subscales 



WWYR Rating Material and Grade 


±0 


±1 


n 


Hypermedia Narratives: Grades 2-3 
Mott & Sumrall (1998) 


.71 


.98 


60 


Pen-and-Paper Narratives: Grades 1-6 
Gearhart, Herman, Novak, & Wolf (1995) 


.46 


.96 


120 


Collections of Pen-and-Paper Narratives: Grades 2-5 
Novak, Herman, & Gearhart (1996) 


.25 


.94 


52 



The percentages of agreement that were revealed in the current study, although higher than those found in 
the Gearhart, Herman, Novak, and Wolf (1995) study, should be considered descriptive information. 
Gearhart et al. remarked that percentages of agreement found for the WWYR should not be interpreted as 
"strong evidence of reliability" (p. 224). Rather, percentages of agreement can be used to help identify the 
existence of widely varying patterns of rater judgments, both across WWYR subscales and across all rater 
pairs. No such widely varying patterns were found in the current study. The limitations of analyses 
involving percentages of agreement analysis were discussed by Abedi (1994), who argued that, although 
percentages of agreement can reveal the existence of widely varying patterns of agreement among raters, 
they can also yield different results from other analyses such as Pearson correlations. 



Pearson Correlations 

Pearson correlations were used to further examine reliability of rater judgments. Pearson correlations 
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averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For 
the five subscales, Theme, Character, Setting, Plot, and Communication, the r values were .59, .55, .49, 
.50 and .50. Table 4 contains the results of the Pearson correlations for WWYR rubric scoring across all 
rater pairs for the current study and for the Gearhart et al. (1995) study. An examination of correlation 
scores for hypermedia narrative productions revealed that interrater reliability for four of the five WWYR 
subscales (Theme, Character, Setting, and Plot) was comparable to the interrater reliability levels found in 
the Gearhart et al. (1995) WWYR reliability study for pen-and-paper-created narratives. For the fifth 
subscale (Communication), however, the correlational coefficient value was .16 higher in the Gearhart et 
al. study than in the current study. Despite the lower value found in the current study for Communication, 
Gearhart et al. related that an average subscale correlation higher than .50 could be considered adequate 
for a rubric such as the WWYR. Table 5 summarizes the comparison of WWYR correlations across all 
subscales for the current study and those found in the literature (Gearhart et al., 1995; Novak et al., 1996). 

Table 4 

Average Pearson Correlations for WWYR Rubric Scoring across 10 Pairs of Raters 



WWYR Rating Material 
and Grade 




Theme 


Character 


Setting 


Plot 


Communication 


Hypermedia Narratives: 
Grades 2-3 


r 


.59 


.55 


.49 


.50 


.50 


Mott & Sumrall, 1998 (n = 
60) 


SD 


.25 


.31 


.25 


.29 


.24 


Pen-and-Paper Narratives 
Grades 1-6 


r 


.64 


.59 


.48 


.57 


.66 


Gearhart et al. (1995) (n = 
120) 


SD 


.10 


.10 


.12 


.14 


.10 



Table 5 

Comparison of WWYR Subscale Correlations: Pen-and-Paper Narratives versus Hypermedia 



Subscale 


Theme 


Character 


Setting 


Plot 


Communication 


Mott & Sumrall (1998) 
Hypermedia 


Samples (n - 60) 


Theme 


- 


.86* 


.79* 


.79* 


.73* 


Character 


- 


- 


.74* 


.74* 


.76* 


Setting 






-- 


.75* 


.68* 


Plot 


- 


-- 






.78* 


Communication 


- 


-- 




- 




Gearhart et al. (1995) 
Pen-and-Paper Narratives 






Samples (n 


= 120) 




Theme 




.83* 


.81* 


.83* 


.86* 


Character 


- 


- 


.82* 


.87* 


.86* 


Setting 




- 


- 


.73* 


.86* 


Plot 


- 


- 


- 


- 


00 


Communication 


- 


- 


- 


- 


- 


Note: *p < .001. 



ERIC 



11 of 17 



12 



3EST COPY AVAILABLE 



http://ecrp.uiuc.edu/v5nl/mott.html 



The WWYR correlations observed in this study, as well as in the Gearhart et al. (1995) study, 
demonstrated that ratings were highly correlated across all subscales. The r values were low in this study 
as well as in the Gearhart et al. (1995) and Novak et al. (1996) studies. However, set guidelines for what 
is an acceptable level of interrater reliability do not exist. Nonetheless, both Gearhart et al. and Novak et 
al., whose studies analyzed holistic scores derived from the combined r values of Theme, Character, Plot, 
Setting, and Communication, argued that r values that fell within the .50 to .70 range were acceptable for 
analytic writing rubrics. In the current study, the interrater reliability for Theme, Character, Plot, and 
Communication subscales fell within the .50 and .59 range, but the level of interrater reliability (r = .49) 
for the Setting subscale did not. It is important to note that, in the Gearhart et al. (1995) study, a low 
coefficient value for the subscale of Setting was also found (r = .48).^ The acceptable interrater 
reliabilities for Theme, Character, Plot, and Communication in this study were comparable to the 
acceptable levels found in the Gearhart et al. (1995) study, and the r values for the Setting subscale in 
both this study and the Gearhart et al. (1995) study were not acceptable (albeit by large-scale standards). It 
is important to note that interrater reliability levels for Theme, Character, and Plot in this study may have 
been lower than the r values in the Gearhart et al. (1995) study because the researcher applied more 
stringent rating procedures in this study. Raters in the Gearhart et al. (1995) study were permitted to 
resolve differences greater than one scale point through discussion, whereas raters in this study were not 
permitted to resolve differences. In the current study, all ratings were included in the final data set. 

The highly correlated rater judgments, along all five WWYR subscales for the current study and for the 
Gearhart et al. (1995) study, provided further evidence of the reliability of WWYR raters’ judgments. The 
true function of a writing rubric is that it "enables raters to apply standard criteria in making judgments 
about the quality of students’ work" (Abedi, 1994, p. 8). Gearhart et al. (1995), Novak et al. (1996), and 
Abedi (1994) argued that highly correlated scores across rubric subscales can be viewed as a positive 
indication that raters’ judgments are being consistently applied. 



Validity 



Developmental Validity 

Developmental validity of WWYR scores was examined via one-way MANOVA conducted on the low-, 
medium-, and high-ability vectors of WWYR subscale scores. The assumption for this analysis was based 
upon the technical qualities of the ITBS — Literacy measure to delineate the developmental literacy levels 
of the children. Results indicated a statistically significant difference between the three ability groups {F 
(2, 36) = 2.59, p = .01). Table 6 provides descriptive statistics, and Table 7 provides an additional 
summary of these results across each of the five WWYR subscales. 

Table 6 

Descriptive Statistics: WWYR Subscales across ITBS Ability Level 





Dependent Variables 


Statistics 


n 


Theme 


Character 


Setting 


Plot 


Communication 


Mean Vectors 














ITBS Ability Level 














Low 


13 


2.31 


1.80 


1.96 


2.10 


2.14 


Medium 


13 


2.80 


2.34 


2.32 


2.66 


2.52 


High 


14 


2.86 


2.60 


2.66 


2.74 


2.77 


Variance-Covariance Matrix 














Theme 




.21 


.15 


.19 


.13 


.11 
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Character 




- 


.23 


.13 


.15 


.13 


Setting 




- 


- 


.19 


.14 


.11 


Plot 




- 


- 


- 


.19 


.14 


Communication 




- 


- 


- 


- 


.16 



Table 7 

Mean WWYR Subscale Scores for Low-, Medium-, and High- Ability Grade-3 Students 



WWYR Subscale 


ITBS NPR/Literacy 
Category 


Mean 

Score 


SD 


n 


F 


Sig 


Theme 


Low 


2.31 


.62 


16 


6.19 


.01 




Medium 


2.80 


.28 


10 




High 


2.86 


.31 


13 


Character 


Low 


1.80 


.50 


16 


10.77 


.01 




Medium 


2.34 


.38 


10 




High 


2.60 


.51 


13 


Setting 


Low 


1.96 


.42 


16 


9.34 


.01 




Medium 


2.32 


.56 


19 




High 


2.66 


.34 


13 


Plot 


Low 


2.10 


.54 


16 


9.28 


.01 




Medium 


2.66 


.34 


10 




High 


2.74 


.34 


13 


Communication 


Low 


2.14 


.47 


16 


9.20 


.01 




Medium 


2.52 


.34 


10 




High 


2.77 


.35 


13 



Tukey HSD tests were conducted on the mean vector scores of the three ability groups for all five WWYR 
subscales to follow up these results. For the WWYR subscale of Theme, low-ability students {M = 2.31, 
SD = .62) received lower scores than both medium-ability students {M - 2.80, SD = .28) and high-ability 
students {M = 2.86, SD = .31). For the WWYR subscales of Character, Setting, Plot, and Communication, 
all differences were significant. Therefore, low-ability students' scores were significantly lower than 
medium-ability students' scores, which were significantly lower than the high-ability students' scores. The 
significant differences revealed between low, medium, and high ITBS groups and the WWYR subscale 
scores provided evidence for the sensitivity of the WWYR to measure the development of students' 
hypermedia/writing competence. The significant results of the one-way MANOVA suggest that raters' 
judgments were evaluating students' skills as message producers (communication through text and other 
meaning-based symbol systems). The one-way MANOVA did not yield results that would enable the 
researcher to describe the degree of relatedness of raters' WWYR judgments and students' ITBS scores. In 
order to describe the relationship between WWYR scores and literacy skill (as measured by the ITBS), 
additional correlational analyses were conducted. 



Concurrent Validity 

The observed Pearson r correlation revealed a positive relationship between students' average WWYR 
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score (averaged across the subscales of Theme, Character, Setting, Plot, and Communication) and their 
ITBS National Percentile Rank (literacy skills score), r = .83, /? > .001 . The positive correlation (r = .83) 
between students' WWYR scores and ITBS scores revealed in this analysis provided evidence for the 
concurrent validity (the degree to which test scores are related to the scores on an already established test) 
of WWYR raters' judgments of hypermedia productions. According to Messick (1992), establishment of 
the concurrent validity of a measure can be a stepping-stone toward establishment of the content-related 
validity (the degree to which scores evaluate the specific domain they were designed to evaluate) of a 
measure. Hence, the developmental and concurrent validities established for WWYR raters' judgments of 
hypermedia productions represented an important initial attempt toward eventually establishing the 
content-related validity of the WWYR when applied to hypermedia productions. 

The strong, positive, linear relationship between ITBS literacy skill scores and WWYR rater judgments of 
hypermedia productions indicated that the hypermedia writing curriculum used in the current study 
involved literacy -based activities. The fact that students in this study expressed themselves through 
hypermedia features, and not solely through text, indicated that students' literacy skill can be enhanced 
through student expression via hypermedia and multimedia features. Table 8 provides additional 
information on the students' utilization of the hypermedia features used in their writing. This finding 
supported the claims of Daiute and Morse (1994), who observed that students who engaged in hypermedia 
writing developed literacy skill through the manipulation of text and other symbols. A weakness of the 
developmental and concurrent validity analyses was that evidence for obtaining the degree to which rater 
judgments of students' hypermedia productions evaluated textual features as well as textual and other 
hypermedia features (audio, hypermedia links, graphics, etc.) could not be determined. 

Table 8 

Frequency of HyperStudio Multimedia Features Used in Students' Hypermedia Narrative Productions* 



Hypermedia/Multimedia Feature 


Grade 

Level 


Button w'ith 
Hypermedia 
Link 


Button 

with 

Audio 


Button 

with 

Video 


Text 

Box 


Graphics 

Text 


Scanned 

Art 


Graphics 
Objects (Clip Art) 


2 (n = 20) 


100% 


81% 


0% 


100% 


45% 


96% 


82% 


o 

11 


100% 


100% 


5% 


100% 


64% 


100% 


100% 


*Note. In three out of the four classrooms where hypermedia/writing occurred, students' use of 
hypermedia/multimedia features was controlled by the teachers. 



Assessment Value and Utility 

The results of this study suggest several important implications for the assessment of students' hypermedia 
products. Having a reliable and valid assessment for evaluating students' hypermedia-based writing serves 
two general purposes: (1) to enhance classroom instruction (value), and (2) to inform, to a lesser extent, 
educational policy (utility). The positive results yielded in this study concerning the reliability and validity 
of the WWYR provide an avenue for teachers to accurately and consistently evaluate their students' 
hypermedia narrative productions by applying the WWYR assessment. The value of an assessment is the 
degree to which it enhances teacher instruction by linking teachers' comments to their instructional 
objectives (Wolf & Gearhart, 1994). Therefore, in order for teachers to properly evaluate both student 
outcomes and the instructional effectiveness of their hypermedia/writing curricula, it is useful for all 
educators to apply a reliable and valid instrument. Furthermore, the positive correlation between the 
students' ITBS literacy skill score and WWYR average score for hypermedia productions indicated that 
students who were engaged in a hypermedia/writing curriculum improved their literacy skills. 

Table 9 

Hypermedia- WWYR 
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Hypermedia 


Theme 


Character 


Setting 


Plot 


Communication 




Explicit- 

Implicit 


FI at- Round 


Backdrop- 

Essential 


Simple- 

Complex 


Context-bound 


Elements 


Didactic- 

Revealing 


Static-Dynamic 


Simple- 

Multifunctional 


Static-Conflict 


Literal-Symbolic 


-Text- 

-Hypertext- 

-Graphic- 

-Audio- 

-Video- 


1 : Not present 
or not 
developed 
through other 
narrative 
elements 


1 : One or two 
flat, static 
characters, with 
little 

relationship 

between 

characters 


1: Backdrop 
setting with little 
or no indication 
of time or place 
("There was a 
little girl. She 
liked candy.") 


1 : One or two 
events with 
little or no 
conflict ("Once 
there was a cat. 
The cat liked 
milk.") 


1: Writing bound to 
context (You have 
to be there) and 
often dependent on 
drawing and talk to 
clarify the meaning 


-Text- 

-Hypertext- 

-Graphic- 

-Audio- 

-Video- 


2: Meaning 
centered in a 
series of list- 
like statements 
("I like my 
mom. And I 
like my dad. 
And I like 
my...") 


2: Some 
rounding, 
usually in 
physical 
description; 
relationship 
between 
characters is 
action driven 


2: Skeletal 
indication of 
time and place 
often held in past 
time ("Once 
there was..."); 
little relationship 
to other narrative 
elements 


2: Beginning 

sequence of 

events, but out- 

of-sync 

occurrences; 

events without 

problem; 

problem 

without 

resolution 


2: Beginning 
awareness of reader 
considerations; 
straightforward style 
and tone focused on 
getting the 
information out 


-Text- 

-Hypertext- 

-Graphic- 

-Audio- 

-Video- 


3: Beginning 
statement of 
theme, often 
explicit and 
didactic ("The 
mean witch 
chased the 
children and 
she shouldn't 
have done 
that.") 


3: Continued 
rounding in 
physical 
description, 
particularly 
stereotypical 
features ("wart 
on the end of 
her nose") 


3: Beginning 
relationship 
between setting 
and other 
narrative 
elements 
(futuristic setting 
to accommodate 
aliens and 
spaceships) 


3: Single, linear 
episode with 
clear beginning, 
middle, and 
end; the episode 
contains a 
problem, 
emotional 
response, 
action, and 
outcome 


3: Writer begins to 
make sense of 
explanations and 
transitions 
("because" and 
"so"); literal style 
centers on 
description 


-Text- 

-Hypertext- 

-Graphic- 

-Audio- 

-Video- 


4; Beginning 
revelation of 
theme on both 
explicit and 
implicit levels 
through the 
more subtle 
things 

characters say 
and do 


4: Beginning 
insights into 
motivation and 
intention that 
drives the 
feeling and 
action of main 
characters often 
through limited 
omniscient 
point of view 


4: Setting 
becomes more 
essential to the 
development of 
the story in 
explicit ways: 
characters may 
remark on the 
setting, or the 
time and place 
may be integral 
to the plot 


4: Plot 
increases in 
complexity with 
more than one 
episode; each 
episode 
contains 
problem, 
emotional 
response, 
action, and 
outcome 


4: Increased 
information and 
explanation for the 
reader (linking ideas 
as well as episodes); 
words more 
carefully selected to 
suit the narrative's 
purpose 


-Text- 

-Hypertext- 

-Graphic- 

-Audio- 

-Video- 


5: Beginning 
use of 
secondary 
themes, often 
tied to 
overarching 
theme, but 
sometimes 
tangential 


5: Further 
rounding (in 
feeling and 
motivation); 
dynamic 
features appear 
in central 
characters and 
between 
characters 


5: Setting may 
serve more than 
one function, and 
the relationship 
between 

functions is more 
implicit and 
symbolic 


5: Stronger 
relationships 
between 
episodes (with 
resolution in 
one leading to a 
problem in the 
next) 


5: Some 
experimentation 
with symbolism 
(particularly 
figurative language), 
which shows reader 
considerations 


-Text- 


6; Overarching 
theme 

multilayered 


6; Round, 
dynamic major 
characters 


6: Setting fully 
integrated with 
the characters. 


6: Overarching 
problem and 
resolution 


6: Careful crafting 
of choices of story 
structure as well as 
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-Hypertext- 


and complex; 


through rich 


action, and 


supported by 


vocabulary 


-Graphic- 


secondary 


description of 


theme 


multiple 


demonstrate 


-Audio- 

-Video- 


themes 
integrally 
related to the 
primary ’ 
themes 


affect, 

intention, and 
motivation 




episodes 


considerate 
orchestration of all 
resources 



Note 

1. A note on correlations averaged across raters: (1) A relatively small number of raters (n = 5) were used in this study 
and the Gearhart, Herman, Novak, and Wolf (1995) study, which may have contributed to the lower r values across 
all subscales. The attenuation of correlational coefficients may be another explanation for the low levels of interrater 
reliability (Gay, 1996). Accordingly, coefficients tend to be lower when a restricted range of values is utilized (e.g,, 
the narrow range of only 3 out of a possible 6 WWYR subscale levels utilized by raters in this study). Thus, the more 
narrow the range of scores utilized by raters, the lower the coefficients. On the other hand, Gearhart et al. argued that 
if the number of raters was statistically increased five-fold, r values in the .50 to .60 range for Theme, Character, 
Setting, Plot, and Communication would be changed to .87, .89, .82, .86, and .89, respectively, Gearhart et al, used 
decision-study (multiplication of sample scores and aggregation of the results) coefficients to determine the number 
of raters needed to attain high reliability coefficients. (2) The r value for the Communication subscale in this study 
was considerably lower than the r value in the Gearhart et al. study (r= .50 versus .66), This sizable disparity in the 
level of interrater reliability may have been the result of the contrasting features of hypermedia-created narrative 
productions versus pen-and-paper-created narratives. The Communication subscale text primarily consisted of 
evaluative prompts designed to guide teachers in the assessment of writing style (see Table 1), Perhaps, in the current 
study, raters solely viewed textual features at the expense of the hypermedia features of graphics, sounds, buttons, and 
scanned art. 
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